Virtualization and dynamic resource allocation aware storage level reordering

ABSTRACT

A system and method for reordering storage levels in a virtualized environment includes identifying a virtual machine (VM) to be transitioned and determining a new storage level order for the VM. The new storage level order reduces a VM live state during a transition, and accounts for hierarchical shared storage memory and criteria imposed by an application to reduce recovery operations after dynamic resource allocation actions. The new storage level order recommendation is propagated to VMs. The new storage level order applied in the VMs. A different storage-level order is recommended after the transition.

BACKGROUND

1. Technical Field

The present invention relates to efficient interoperation of computingtechnologies, and more particularly to systems and methods for storagelevel reordering for virtual and dynamic resource allocation formemory-intensive applications.

2. Description of the Related Art

Emerging Internet-scale service applications operate on large amounts ofdata that are maintained in memory to achieve high throughput and offerlow response time guarantees to users. Key catalyzers to this trend havebeen the significant decrease of memory cost, increase of memorycapacity in machines and development and adoption of in-memorydistributed storage technologies such as memcached, XAP and ObjectGrid™As a result, In-Memory Data Grids (IMDGs) have become the cost- andperformance-effective solution for such vast-scale and distributed dataprovisioning. Thus, many Web-scale businesses, such as Facebook™ andTwitter™, rely on some form of such memory-backed data grids to operatewhile scaling to a large number of users.

Facebook™, for example, stores all its indexing information on amemcached to speed up the access to large objects from backend storageservers in a timely manner. Other non-traditional data orientedapplications, e.g., fraud detection, rely on IMDG to collect and analyzeoperational data in an on-line fashion.

Virtualization technologies are becoming more important in large-scaledata center and cloud infrastructures. In addition to the simplifiedmanagement and provisioning benefits, these virtualized environmentsenable dynamic optimizations to the cloud operation by dynamicallyreallocating distributed resources to the running virtualized entities.By repositioning and reprovisioning virtual machines (VMs), virtualizedenvironments provide agile runtime response to the dynamically-varyingrequirements of data centers. This alleviates the inefficiencies of theexisting statically-allocated information technology (IT) infrastructuremodel due to the severe over-provisioning and inefficientunderutilization of computing resources.

While both of these trends have proven to be effective exclusively intheir domains and continue to gain increasing momentum, they exhibit astriking conflict in their characteristics that can hamper theireffectiveness under joint operation in a cloud environment. Morespecifically, on one hand, IMDG solutions benefit from exploiting largeamounts of random access memory (RAM) to manage data and therefore, theytend to heavily exercise memory for effective operation.

On the other hand, dynamic VM allocation in virtualization performs bestby transferring as little live state as possible across physicalentities during VM reallocation. It is well known that there is a strongpositive correlation between the amount of live state a VM has and themigration overheads it can experience. The main contributors to the livestate are the amount of ‘active memory’ or the working set size of theVM, the rate at which the pages are dirtied in memory and the overallfootprint of the VM. Therefore, their dynamic response efficiency isinversely affected while allocating VMs with heavy memory usage. Thishas at least two detrimental effects: (1) higher performance overheadperceived by the applications using IMDG nodes as the response timesincrease under dynamic allocation conditions; and (2) energy overheadsdue to extensive copying and recopying of live state, as well as higherresource and link usage.

SUMMARY

Migration latency depends on an active footprint of a virtual machine(VM). For example, a 16× increase in active memory leads to an almost32× increase in the total migration overhead. Corresponding CPU andmemory usage during these migrations show an average of 40% single-coreCPU overhead while migrating a VM from a source host, and around 20% CPUoverhead while migrating a VM to a destination host. In both directions,the migration traffic can saturate 1 Gbps links throughout migration.Quite significant degradations can also be observed in the applicationperformance, such as service times, depending on the applicationcharacteristics and the level of overcommitment on the host. Therefore,an efficient migration-aware state management from both resourcemanagement technology and application performance management technologyperspectives is needed. The present principles provide a virtualization-and dynamic-resource-allocation-aware storage level reordering method(VSLR) for memory-intensive applications, as an in-memory data grid(IMDGs), for highly-effective interoperation of these two growingtechnologies. With both technologies becoming widely adopted in similarlarge-scale computing and cloud-based infrastructures, suchinteroperation provides cooperation between these technologies andothers.

A system and method for reordering storage levels in a virtualizedenvironment includes identifying a virtual machine (VM) to betransitioned and determining a new storage level order for the VM. Thenew storage level order reduces a VM live state during a transition, andaccounts for hierarchical shared storage memory and criteria imposed byan application to reduce recovery operations after dynamic resourceallocation actions. The new storage level order recommendation ispropagated to VMs. The new storage level order is applied in the VMs. Adifferent storage-level order is recommended after the transition.

A method for reordering storage levels in a virtualized environmentincludes identifying a virtual machine (VM) to be reprovisioned orrepositioned to a new location; retrieving profiling informationregarding usage and availability of multiple storage levels and devices;and determining a new storage level order for the VM using the profilinginformation that reduces a VM live state during a migration transfer orreprovisioning and accounts for at least shared storage memory, VMvirtual disk locations and service level agreement (SLA) requirementsimposed by an application to reduce recovery operations after dynamicresource allocation actions. The new storage level order recommendationis propagated to VMs, and the new storage level order is applied to theVMs. An original storage-level order is reverted back to after thereprovisioning or repositioning actions. Resource allocation is computedfor a current cluster state to determine whether further reprovisioningor repositioning of VMs is needed. The method is continued until furtherVM reprovisioning or repositioning is complete.

A system for dynamically reordering storage levels for applications invirtualized systems includes a virtual machine to be reprovisioned orrepositioned, stored in a memory storage device. A virtualized resourcemanager (VRM) is configured to recommend a new storage level order in amemory media hierarchy for the VM that reduces a VM live state duringreprovisioning or migration, accounts for storage levels based uponprofiling information, and reduces recovery operations after dynamicresource allocation actions. A virtualization- anddynamic-resource-allocation-aware storage level reordering (VSLR) moduleis configured to propagate and apply the new storage level orderrecommendation to appropriate memory intensive VMs such that reorderingmitigates the performance overhead of VM reprovisioning or repositioningactions.

These and other features and advantages will become apparent from thefollowing detailed description of illustrative embodiments thereof,which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

The disclosure will provide details in the following description ofpreferred embodiments with reference to the following figures wherein:

FIG. 1 is a block/flow diagram showing a computing environment with anillustrative migration of a virtual machine (VM) in accordance with thepresent principles;

FIG. 2 is a block/flow diagram showing a system/method for repositioningVM's in accordance with the present principles; and

FIG. 3 is a block/flow diagram showing a system/method forintercooperation between technologies for executing a change in avirtualized environment in accordance with another embodiment.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present principles provide a system having multi-level data storageincluding a low latency/high bandwidth storage medium, e.g., RAM, and ahigher latency/lower bandwidth storage medium, e.g., disk, which operatein a hierarchical fashion. In the context of, e.g., in-memory data grid(IMDG) appliances, a storage level with best latency/size ratio isconsidered primary and hence has the highest priority in the hierarchy.For example, IMDG technologies, such as ObjectGrid™, operate primarilyon RAM and rely on disk storage to respond to overflow conditions.

A virtualization- and dynamic-resource-allocation-aware storage levelreordering (VSLR) module permits applications such as IMDGs to work inconjunction with a system virtual resource manager (VRM) to mitigateoverhead resulting from virtual machine (VM) migration by effectivelyreducing live state data of a hosting VM to be migrated. Reducing theactive state of these applications can lead to several orders ofreductions in dynamic resource allocation overheads.

To do this, IMDG nodes reorder the storage hierarchy levels uponmigration of the hosting VM by demoting RAM to secondary storage andpromoting lower storage levels in the hierarchy. In other words, IMDGnodes are capable of interacting with different levels of the storagehierarchy throughout their life cycle as dictated by a VSLR managementsystem. Furthermore, in VSLR the order is determined by the application(IMDG) and the resource manager in conjunction so that application andthe system requirements are both met.

By adopting VSLR, a Cloud provider, for example, can effectively managethe provisioning and repositioning of resources while meeting therequirements of Cloud services that rely heavily on IMDG solutions tomanage distributed data under high response time constraints. No knownsolution addresses the problem of managing services such as IMDG in avirtual environment.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing. Computer program code for carrying out operations foraspects of the present invention may be written in any combination ofone or more programming languages, including an object orientedprogramming language such as Java, Smalltalk, C++ or the like andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer, other programmable data processing apparatus, orother devices to cause a series of operational steps to be performed onthe computer, other programmable apparatus or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the FIGS. illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Referring now to the drawings in which like numerals represent the sameor similar elements and initially to FIG. 1, a system 100 and method fora cooperative technology environment is illustratively depicted. System100 illustratively includes a virtual environment having five maincomponents: physical machines 102, 104, virtual machines 106, VirtualResource Manager (VRM) 110, Virtualization- anddynamic-resource-allocation-aware storage level reordering (VSLR) module112 and Storage Resource Manager (SRM) 114. Applications 116 are hostedby individual VMs 106, and physical machines 102, 104 may host multipleVMs 106. Each VM 106 has a share of resources (network, memory and CPU)allocated at boot time to the VM 106 and shares resources with other VMs106 co-hosted in the same physical machine 102, 104. The VSLR system 100illustratively shows two physical machines: a Home machine (Home) 102and a Target machine (Target) 104, hosting multiple VMs 106. Home 102hosts a VM 106 (VM1) with an IMDG node 118 that is to be migrated toTarget 104.

SRM 114 is responsible for monitoring storage usage across the storagehierarchy in the system 100. SRM 114 builds profiling information forstorage and storage devices and provides the profiling information tothe VRM 110. The VRM 110 is responsible for making decisions regardingthe repositioning (migration) and reprovisioning of virtual machines106; coordinating with the VSLR module 112 and SRM 114, when needed, anddetermining a new storage level order for an IMDG (118) to use when itshosting VM (106) has been selected for potential repositioning. VRM 110and hypervisors 120 are the only components that need to be aware ofdynamic resource management decisions. VSLR modules 112 and the VMs 106only need to receive the dynamic storage level reorderingrecommendations without necessarily being aware of the underlyingvirtualization layer.

VSLR module 112 plays a mediator role between the IMDG nodes 118 and theVRM 110. VSLR module 112 can be implemented as an agent or as anauxiliary VM 106 in each host 102 or 104. VSLR module 112 is responsiblefor informing the IMDG nodes 118 of the storage level reorderingrecommendations and reporting to the VRM 110 employed storage levelorders by each node.

During a migration, system 100 performs the following illustrativefunctions as indicated by circled numbers in FIG. 1: (1) VRM 110 informsVSLR module 112 of VM1 being selected for migration and suggests a newstorage level order for the IMDG node 118 hosted in VM1 (disk only inthis case). (2) VSLR module 112 propagates the new storage level orderrecommendation to the IMDG container 122 in VM1; (3) IMDG 118 followsthe new order and enters Allocation-Aware Mode and writes to a disk 126for all future incoming transactions; (4) Migration of VM1 starts andfinishes; (5) IMDG 118 optionally enters into Recovery Mode and syncsHome machine 102 and Target machine 104, if necessary, by applyingtransactions committed during migration; VRM 110 recommends reversingthe storage level order to Normal Mode via VSLR module 112.

The present embodiments are particularly useful for any in-memoryintensive application, e.g., memcached, ObjectGrid™, etc. However, thepresent implementation varies depending on the application. The aboveimplementation is based on the context of ObjectGrid™ (IBM IMDG).ObjectGrid™ has built-in capabilities to interact with different storagedevices 130 through a loader module 124 embedded in the physical devices102, 104 (e.g., server) and hence, it is an ideal illustrative choicefor this disclosure. Other configurations and contexts are alsocontemplated.

Whenever a VM has been marked for repositioning, the VRM 110 informs theVSLR module 112 in the Home machine 102 and suggests a new storage levelorder following profiling information obtained from the SRM 114 (seecircled 1). The VSLR module 112 then informs the IMDG node 118 hosted inthe selected VM of the recommended storage order.

As soon as the VRM 110 selects a given VM 106 hosting an IMDGapplication 116 for repositioning, the VRM 110 obtains storage profilinginformation from the SRM 114. For example, VRM 110 may obtain thecurrent read/write request rate of the local disk 126 or the currentusage of the flash memory card attached to the Home machine 102. The VRM110 uses this information to suggest to the corresponding VSLR module112, a new storage level order for the IMDG node 118 to use. Forexample, assuming that the Home machine 102 has access to a remoteshared storage 130 and local storage 126, the VRM 110 may suggest remotestorage as primary storage if its effective bandwidth is above a certainthreshold and use local disk 126 or main memory (not shown) to handleoverflow conditions.

One strength of the present principles stems from performing thisstorage level reordering in a virtualization-aware manner. In manyexisting virtualized infrastructures a shared, external storage, in theform of a storage area network (SAN) or network-attached storage (NAS),is used to back the virtual hard disks of VMs 106. A virtual hard diskof the IMDG VM 118 that is to be migrated can likely be residing on suchan external storage (130). In this case, promoting the VM's disk as theprimary storage alleviates the need to retransfer any live state as allfurther updates are directly visible to the VM after migration as theVM's disk location is unchanged and shared across hosts. In thisoperation, VSLR module 112 helps efficiently manage IMDGs 118 invirtualized environments by dramatically mitigating the overhead ofdynamic resource allocation actions of the underlying virtualizationmanagement layer. The overall IMDG operation continues with minimaldisruption other than the reordering of storage levels. Nonetheless, asdemonstrated by the present inventors, shared storage is not a brickwall necessity for dynamic resource allocation. More importantly,applications with strong data-locality requirements—such as Mapreduce™and data-analytics—benefit greatly from using VMs with attached storage.Under such conditions, VMs with detached storage can be migrated acrosshosts.

For generally accommodating these cases, the present embodimentsintroduce a separate IMDG operation mode for recovery after resourceallocation. For dynamic reordering of storage level, we may assume thatthe IMDG node 118 adopts the order recommended by the VSLR module 112.Nevertheless, this assumption can be easily relaxed to accommodate formore intelligent application-based resource management techniques.Throughout its life cycle an IMDG node can switch between multiplemodes, e.g.: Normal Mode, Transition Mode and Recovery Mode. Other modescan be designed and implemented, as needed.

In Normal Mode an IMDG 118 operates with the storage level orderdetermined at boot time. In other words, user transactions are effectedinto IMDG main memory only and secondary storage is used underoverflowing conditions. This corresponds to the normal operation of anIMDG node. The IMDG node 118 enters into the Transition Mode as soon asit obtains the new storage level order from the VSLR module 112. Duringthis mode the IMDG node 118 is enabled to follow a different storagelevel order—as suggested by the VRM 110—to effectively write-through astorage device that may have a lower order in the Normal Mode. Thisresults in a reduction on the amount of live state data in the VMhosting the IMDG node 118, and thus drastically reduces the overhead ofdynamic resource allocation.

An IMDG node 118 can switch to Recovery Mode after a Transition Modetriggered during VM migration, i.e., after the VM's live state data hasbeen migrated to the Target machine 104. This mode includes applying allthe transactions committed while the IMDG 118 was in Transition Mode tothe memory space assigned to the IMDG node in the Target machine 104.This is not a generally necessary mode if the transactions committed inthe transition mode are already persisted to the VM 106 as in the caseof shared storage model. Overall, the present principles describe aunique and effective method for efficient interoperation ofvirtualization resource management and memory intensive applications,with specific emphasis on IMDGs although other system may be employed.By incorporating the described VSLR module 112 into their jointoperation and enabling dynamic reordering of the storage levels, thepresent embodiments provide an efficient approach to manage twoimportant cloud computing technologies that otherwise exhibit competingcharacteristics. The described systems and methods transform suchcompeting characteristics into a clearly-defined cooperative operation.

In particular, efficient migration-aware state management from bothresource management and application performance management perspectivesis provided by employing the VSLR module 112 in accordance with thepresent principles. The VSLR module 112 is especially useful formemory-intensive applications for highly-effective interoperationbetween different technologies.

Referring to FIG. 2, a method for implementing a change usingstorage-level orders is shown in accordance with one illustrativeexample. In block 202, a VRM marks VMs for repositioning. In block 204,the VRM obtains storage profiling information from the SRM. The SRMbuilds profiling information regarding the usage/availability of themultiple storage levels and devices in the system to obtain profilinginformation from the storage layer. For example, it may obtain effectivebandwidth of local disks and network storage, memory usage of flashmemory cards in physical machines, etc. Methods to obtain this type ofinformation may take a plurality of forms. For example, existingproducts such as VMware may include built-in modules to obtain thisprofiling information with minimum overhead.

In block 206, VRM provides a new storage-level order recommendation forIMDG VMs. In block 208, VRM informs IMDGs in VMs of the newstorage-level order through a VSLR module. In block 210, IMDGs apply thenew storage-level order. In block 212, VRM carries out repositioningactions. Repositioning of VMs can be triggered due to many reasons. Forexample, maintenance tasks, e.g., shutting down a physical machine toperform remediation and upgrade tasks, dynamic resource managementactions to mitigate dynamically-emerging resource contention issues,etc. There exist multiple techniques for repositioning VMs.

In block 214, VRM may recommend reverting to an original storage-levelorder through VSLR the module. In block 216, VRM waits until a nextresource management period. In block 218, VRM computes an optimalresource allocation for a current cluster state or memory hierarchy. Inblock 220, a determination as to whether any VM repositioning is neededis made. If yes, the path goes to block 202, and if no, the path goes toblock 216.

Referring to FIG. 3, a method for reordering storage levels in avirtualized environment, which executes the repositioning of VMs moreefficiently especially for memory intensive applications isillustratively depicted in accordance with one exemplary embodiment. Thememory intensive application may include memory intensive applicationsin the form of in-memory data grids (IMDGs).

In block 302, a virtual machine (VM) to be repositioned to a newlocation and/or reprovisioned is identified or selected. In block 304, anew storage level order for the VM is determined that reduces a VM livestate during a migration transfer and accounts for: shared storagememory, VM virtual disk locations and service level agreement (SLA)requirements imposed by an application, etc. to reduce recoveryoperations after dynamic resource allocation actions. Otherconsiderations may also be employed in making the recommendation. In oneembodiment, in block 307, storage level conditions are evaluated by avirtualization resource manager (VRM) using a storage resource manager(SRM), which provides profile information for different storage levels.In block 308, the VRM decides on dynamic resource allocation actions,and produces storage level reordering recommendations for different VMs.

In block 310, the new storage level order recommendation is propagatedto VMs and the new storage level order is applied in the VMs. In block312, a virtualization- and dynamic-resource-allocation-aware storagelevel reordering (VSLR) module mediates between application nodes (e.g.,the IMDG nodes) and the VRM to propagate VRM recommendations to eachnode, monitor node actions and report current behavior to VRM.

In block 314, application nodes (e.g., IMDGs) include distinct operationmodes for normal operation, transition mode when a storage level isreordered, and recovery mode to return to normal operation after thetransition mode. The modes are provided to ensure efficient transitionactions (e.g., repositioning or reprovisioning). In block 316, the nodes(e.g., IMDGs) may employ at least one additional intelligentself-regulation mechanism to dynamically choose a storage level orderbased on VSLR recommendations and local optimality criteria. The IMDGnode or other node can take the recommendation of the VRM or mayoverride the recommendation in accordance with selected criteria. Forexample, the VRM may recommend the nodes to set RAM-based storage to thehighest level (i.e., primary), however in accordance with local CPU ormemory usage monitoring, the node may select to disk-based storage. Bymeans of simple feedback control mechanisms the node can also leverageinformation regarding the performance perceived by the application(e.g., response time, throughput). For instance, the VRM may recommend a2-level storage hierarchy to the nodes including flash-based anddisk-based storage as primary and secondary level respectively. If theapplication has reported poor throughput, the nodes may favor a moreresponsive storage ordering such as the one having RAM-based andflash-based storage, respectively.

In block 318, a different storage-level order is recommended after therepositioning actions. This may include reverting back to the originalstorage-level order after the repositioning actions, or adopting a newstorage level order. In block 320, a next VM to be transitioned isconsidered until all VMs selected or identified for transitions arereached.

Having described preferred embodiments for virtualization and dynamicresource allocation aware storage level reordering (which are intendedto be illustrative and not limiting), it is noted that modifications andvariations can be made by persons skilled in the art in light of theabove teachings. It is therefore to be understood that changes may bemade in the particular embodiments disclosed which are within the scopeof the invention as outlined by the appended claims. Having thusdescribed aspects of the invention, with the details and particularityrequired by the patent laws, what is claimed and desired protected byLetters Patent is set forth in the appended claims.

1. A method for reordering storage levels in a virtualized environment,comprising: identifying a virtual machine (VM) to be transitioned;determining a new storage level order for the VM that reduces a VM livestate during a transition, and accounts for hierarchical shared storagememory and criteria imposed by an application to reduce recoveryoperations after dynamic resource allocation actions; propagating thenew storage level order recommendation to VMs and applying the newstorage level order in the VMs; and recommending a differentstorage-level order after the transition.
 2. The method as recited inclaim 1, wherein the application includes memory intensive applicationsin the form of in-memory data grids (IMDGs).
 3. The method as recited inclaim 2, wherein the IMDGs have distinct operation modes for a normalmode, a transition mode when a storage level is reordered, and arecovery mode to return to normal operation after the transition mode.4. The method as recited in claim 1, wherein determining a new storagelevel order includes evaluating storage level conditions by avirtualization resource manager (VRM) using a storage resource manager(SRM), which provides profile information for different storage levels.5. The method as recited in claim 4, wherein the VRM decides on dynamicresource allocation actions, and produces storage level reorderingrecommendations for different VMs.
 6. The method as recited in claim 5,further comprising: mediating between the IMDG nodes and the VRM using avirtualization- and dynamic-resource-allocation-aware storage levelreordering (VSLR) module, wherein the VSLR module propagates VRMrecommendations to each IMDG node and monitors IMDG actions and reportscurrent behavior to the VRM.
 7. The method as recited in claim 6,wherein the IMDGs employ at least one additional intelligentself-regulation mechanism to dynamically choose a storage level orderbased on VSLR recommendations and local optimality criteria.
 8. Acomputer readable storage medium comprising a computer readable programfor reordering storage levels in a virtualized environment, wherein thecomputer readable program when executed on a computer causes thecomputer to perform the steps of: identifying a virtual machine (VM) tobe transitioned; determining a new storage level order for the VM thatreduces a VM live state during a transition, and accounts forhierarchical shared storage memory and criteria imposed by anapplication to reduce recovery operations after dynamic resourceallocation actions; propagating the new storage level orderrecommendation to VMs and applying the new storage level order in theVMs; and recommending a different storage-level order after thetransition.
 9. The computer readable storage medium as recited in claim8, wherein the application includes memory intensive applications in theform of in-memory data grids (IMDGs).
 10. The computer readable storagemedium as recited in claim 9, wherein the IMDGs have distinct operationmodes for a normal mode, a transition mode when a storage level isreordered, and a recovery mode to return to normal operation after thetransition mode.
 11. The computer readable storage medium as recited inclaim 8, wherein determining a new storage level order includesevaluating storage level conditions by a virtualization resource manager(VRM) using a storage resource manager (SRM), which provides profileinformation for different storage levels.
 12. The computer readablestorage medium as recited in claim 11, wherein the VRM decides ondynamic resource allocation actions, and produces storage levelreordering recommendations for different VMs.
 13. The computer readablestorage medium as recited in claim 12, further comprising mediatingbetween the IMDG nodes and the VRM using a virtualization- anddynamic-resource-allocation-aware storage level reordering (VSLR)module, wherein the VSLR module propagates VRM recommendations to eachIMDG node and monitors IMDG actions and reports current behavior to theVRM.
 14. The computer readable storage medium as recited in claim 13,wherein the IMDGs employ at least one additional intelligentself-regulation mechanism to dynamically choose a storage level orderbased on VSLR recommendations and local optimality criteria.
 15. Amethod for reordering storage levels in a virtualized environment,comprising: identifying a virtual machine (VM) to be reprovisioned orrepositioned to a new location; retrieving profiling informationregarding usage and availability of multiple storage levels and devices;determining a new storage level order for the VM using the profilinginformation that reduces a VM live state during a migration transfer orreprovisioning and accounts for at least shared storage memory, VMvirtual disk locations and service level agreement (SLA) requirementsimposed by an application to reduce recovery operations after dynamicresource allocation actions; propagating the new storage level orderrecommendation to VMs and applying the new storage level order in theVMs; reverting back to an original storage-level order after thereprovisioning or repositioning actions; computing resource allocationfor a current cluster state to determine whether further reprovisioningor repositioning of VMs is needed; and continuing the method untilfurther VM reprovisioning or repositioning is complete.
 16. The methodas recited in claim 15, wherein the application includes memoryintensive applications in the form of in memory data grids (IMDGs),wherein the IMDGs have distinct operation modes for a normal mode, atransition mode when a storage level is reordered, and a recovery modeto return to normal operation after the transition mode.
 17. The methodas recited in claim 15, wherein determining a new storage level orderincludes evaluating storage level conditions by a virtualizationresource manager (VRM) using a storage resource manager (SRM), whichprovides profile information for different storage levels, wherein theVRM decides on dynamic resource allocation actions, and produces storagelevel reordering recommendations for different VMs.
 18. The method asrecited in claim 17, further comprising: mediating between the IMDGnodes and the VRM using a virtualization- anddynamic-resource-allocation-aware storage level reordering (VSLR)module, wherein the VSLR module propagates VRM recommendations to eachIMDG node and monitors IMDG actions and reports current behavior to theVRM.
 19. The method as recited in claim 18, wherein the IMDGs employ atleast one additional intelligent self-regulation mechanism todynamically choose a storage level order based on VSLR recommendationsand local optimality criteria.
 20. A system for dynamically reorderingstorage levels for applications in virtualized systems, comprising: avirtual machine to be reprovisioned or repositioned, stored in a memorystorage device; a virtualized resource manager (VRM) configured torecommend a new storage level order in a memory media hierarchy for theVM that reduces a VM live state during reprovisioning or migration,accounts for storage levels based upon profiling information, andreduces recovery operations after dynamic resource allocation actions;and a virtualization- and dynamic-resource-allocation-aware storagelevel reordering (VSLR) module configured to propagate and apply the newstorage level order recommendation to appropriate memory intensive VMssuch that reordering mitigates the performance overhead of VMreprovisioning or repositioning actions.
 21. The system as recited inclaim 20, wherein the memory intensive VMs are in the form of in memorydata grids (IMDGs).
 22. The system as recited in claim 21, wherein theIMDG includes distinct operation modes including a normal mode, atransition mode when the storage level is reordered, and a recovery modeto return to normal operation after the transition mode.
 23. The systemas recited in claim 21, wherein the VSLR module acts as a mediatorbetween an IMDG node and the VRM such that the VSLR module propagatesVRM recommendations to each IMDG node and the VSLR module monitors IMDGactions and reports current behavior to the VRM.
 24. The system asrecited in claim 21, wherein the IMDG node is configured to employ aself-regulation mechanism to dynamically choose its storage level orderbased on VSLR recommendations and its local optimality criteria.
 25. Thesystem as recited in claim 20, further comprising: a storage resourcemanager (SRM) configured to provide the profile information fordifferent storage levels to assist the VRM in making itsrecommendations.